Software based content processing solutions are deployed on e-mail servers or gateway devices such as firewalls or proxies. These software solutions are easily updated and upgraded with new information, algorithms or techniques. The problem with these software implementations is that they are both too slow for deployment in the actual networks (where they would be in line with traffic) and are not deterministic enough, which can add significant jitter.
Implementing such complex processing capabilities can be done in hardware which solves the problems of performance and jitter, but such hardware solutions usually can not be upgraded with such ease. With software solutions a new set of instructions is executed on an invariant platform, but where a hardware solution is re-designed the verification and testing process required is usually prohibitive in terms of time (e.g. responses to new threats must be made available within minutes, and new protocols or functions may be required within weeks or months). Indeed, hardware design cycles can sometimes take years.
The latest high-speed processing solutions usually incorporate software and hardware elements together. Software elements must execute on a CPU of type RISC, CSIC or DSP, none of which is optimised for content processing. Hardware solutions are collections of transistors or gates synthesised from high-level code, where a change in code requires a complete re-synthesis where the entire device changes, requiring a stringent and time consuming validation cycle.
Software approaches to the problem are inherently serial in operation. Regular expression matchers must be run one after the other and are therefore relatively slow. Signature matchers are faster, since a corpus of signatures is compiled to produce a single optimised state machine, but these generally require a final byte-by-byte comparison to establish an exact match. A software-only approach based on a general-purpose microprocessor must generally perform checks on the amount of content remaining etc., during the match process such that many instructions must be executed per byte of content passed through the system.
Existing hardware approaches are decoupled from the software that drives them. For example, an ldt Network Search Engine processes each network packet and delivers a digest of predefined fields in the packet to the associated processing element. Furthermore, these engines are essentially packet-based and do not address the needs of products working above the packet level on a reassembled content byte stream containing an OSI Layer 5 (or above) protocol.
A solution is sought that would enable complex processing to be performed at high enough speeds and with sufficiently low latency that it can be incorporated into network devices which sit in line with network traffic.